THE GUIDE TO PERFORMANCE STANDARDS FOR INSTRUCTORS OF CREW RESOURCE
MANAGEMENT TRAINING IN COMMERCIAL AVIATION
GUIDE TO PERFORMANCE STANDARDS FOR INSTRUCTORS OF CREW RESOURCE MANAGEMENT TRAINING IN COMMERCIAL AVIATION
1. The aims of the survey are to: establish the extent of the
use of the Guide to Performance Standards for Instructors of Crew Resource
Management (CRM) Training in Commercial Aviation (the Guide); its
credibility within the aviation industry; and hence determine its validity as a
tool in the accreditation of CRM instructors.
2. The Guide was produced in September 1998 by the Royal Aeronautical Society (RAeS) Human Factors Group (HFG), in concert with the UK Civil Aviation Authority (CAA) and the Aviation Training Association (ATA). The product of a 2-year research programme that included two industry workshops, the Guide was made available free from the printers or via the RAeS HFG website and has been received by more than 1550 people in 67 countries.
3. In the absence of other material, the Guide was produced in response to a need for standards of competence for CRM instructors in the UK. Its use was encouraged by the CAA through UK Aeronautical Information Circular (AIC), 114/1998 (Pink 178) dated 6 October, which states that “the Authority expects that all who give instruction in CRM and associated subjects will meet the performance criteria indicated by the performance standard Guide” (Paragraph 2.1.3).
5. To ensure impartiality and confidentiality, the survey was conducted by an independent aviation consultant, Mr Pieter Hemsley, with costs underwritten by the ATA, the competence standards body for the UK aviation industry. Throughout the project, the consultant has given assurances that all data has been collated anonymously and used solely to validate the Guide. There is neither attribution of comments from individual respondents nor identification of any views from specific operators.
6. The AIC forecast that “the Authority will seek to establish a means of CRM instructional accreditation, after further consultation with industry and collaboration with the JAA”. To take forward such work, an independent CRM Advisory Panel has been established to provide, inter alia, advice to the CAA to ensure the highest standards of HF and CRM education and practice. This Panel is continuing to work with the industry in a pragmatic way to ensure a robust, credible CRM trainer accreditation architecture is brought to maturity. The report will be made available to the Panel to inform its work.
7. The survey was targetted at all UK Air Operator Certificate (AOC) Holders and all other UK training providers and individual recipients of the Guide, including those who had attended the industry workshops. To avoid bias, RAeS HFG and CRM Advisory Panel members were specifically excluded. During the first mailing in June 2000, 477 copies were distributed to AOCs (230) and others (247). A further 52 were despatched in November 2000 to those who had taken copies of the Guide during the summer, giving a total population of 529. Thus, the target group was wider than that which will be subject to formal accreditation in the first instance and the survey therefore captures some - from the flying training schools within the licensing regime and from non Public Transport (PT) fields such as General Aviation (GA) - who may be subject to the same framework at a later date; their views are considered equally valid.
8. A total of 92 replies (17% of population) was received, of which 32 (35% of responses or 6% of population) reported to have not seen the Guide, ie, 60 (65% and 11% respectively) had sighted the document. Although this figure is somewhat disappointing, the numbers are sufficient to be statistically representative and the views expressed may be taken as typical of those throughout the industry who have read and used the document. Further, we can surmise that the Guide is not being used as widely as its distribution would suggest.
9. The silent majority who did not reply had a further opportunity when the survey was advertised in CAA Flight Operations Department Communication 13/2000 dated 20 November; however, no responses were received following this stimulation. Neither were any received after the survey was posted on the RAeS HFG website. The reasons for not replying will be many but are likely to include: the fact that some will not yet be subject to the proposals, for example, in the flight crew licensing regime; a lack of experience or expertise in CRM leading some to not have the confidence to express their views; a feeling of resignation that the proposals will proceed regardless of comment; and apathy in the face of yet another survey.
10. Despite their relatively small number, many respondents were highly experienced, senior workers within the industry. Most pilots held several qualifications, had a great many flying hours and a significant number were also responsible for CRM training. The following key statistics emerged (note that not all figures give the same totals as respondents were sometimes inconsistent in completing questions or omitted data; all results are at Appendix 2, Tables 1 to 10, further referenced by survey question number):
Of 72 crew reporting: 60 were fulltime and 6 freelance (Question 5, Table 3); and 6 retired but still active as consultants or trainers;
There were: 57 captains (79%), including one female (Question 4, Table 2);
32 commanders conducting line checks (44%) (Table 8);
37 training captains (51%) (Table 8);
32 Type Rating Instructors (TRIs) (44%) (Table 8);
34 Type Rating Examiners (TREs) (47%) (Table 8).
In addition, of the total 92 respondents, 68% gave personal views, whilst 32% expressed official company policy (Question 3, Table 1);
39 were groundschool CRM instructors (42%) (Table 8);
13 were LOFT instructors (14%) (Table 8);
31 managers responsible for CRM training (34%) (Table 8);
15 conducting MCC training (16%) (Table 8);
21 employed in a Type Rating Training Organisation (TRTO) (23%) and 15 in a Flying Training School (FTS) (16%) (Table 8);
47 were from Public Transport operations (51%) (Table 9); and,
6 were Flight Operations Inspectors (FOIs) from the Safety Regulation Group of the CAA (7%) (Table 10).
The experience level is further given by:
The mean total flight time for all aircraft types (of 72 crew), which was almost 9900 hours (aeroplane 9700 and helicopter 10300), with a range of 17000 hours, ie, between 20000 and 3000 (Question 5, Table 6);
Mean years flying was 29 (71 crew), with a range of 35 years, ie, between 10 and 45 years (Question 5, Table 7);
Crew were split 77% Aeroplane and 23% Helicopter, the latter including police; very many aircraft types were operated (Question 5, Tables 4 and 5).
The 32 ‘other’ respondents (ie, neither a captain nor a first officer) included: 2 professors of psychology; 3 doctors of medicine or psychology; 3 flight engineers; 3 cabin crew; and one former flight navigator (Question 6, Tables 8, 9 and 10);
Non-crew included 5 females, ie, there were 6 female respondents in toto (7%) (Question 4, Table2).
11. Such depth of experience gives strong grounds for optimism that the data given and the views expressed have credibility with their peers and with the industry at large.
Extent of the Use of the
Guide
12. The Guide has been widely distributed although indications are that only some 33 AOC Holders and training providers had seen the document. Given that it has also been extensively publicised since it was produced in September 1998 – through the AIC that went to all PT AOC Holders and on the RAeS HFG website – and has been distributed free, the take-up by industry has been limited. In part, this could stem from the view of some reporting that their companies do the minimum to meet only CRM legislated requirements and, until its use is mandated, it will be largely ignored (Table 12 et al).
13. Examining the 3 contexts in the Guide (Question 7, Table 11), 42 persons (47%) were involved in Groundschool/Classroom Instruction, with 25 (28%) in Checks Instruction/Examination and 23 (25%) in Aircraft, Simulator and LOFT.
14. Of those who had seen the Guide, a high proportion (67%) (Question 8, Table 12) was using it as a personal reference, not surprising given the large number of respondents holding training appointments. However, use for teams was less widespread at 35% and few – just 25% - were using the document for developing standards for CRM instructors, whilst only 18% were using it for developing assessment criteria. In addition, the Guide was in use for the assessment of CRM/LOFT trainers in the simulator and for the evaluation of CRM skills.
15. Interestingly, it was used in other arenas, such as: anaesthesia crisis resource management; the offshore oil industry; and clinical risk management.
Customer Satisfaction
16. A majority of the Guide recipients (83%) completed Question 9 relating to satisfaction with the document (Table 13). There was a high degree of approval:
66% ‘Agree’ or ‘Strongly Agree’ to the statement that it is ‘User-friendly’;
74% ‘Agree’ or ‘Strongly Agree’ that it is ‘Clear and unambiguous’;
90% ‘Agree’ or ‘Strongly Agree’ that it is ‘Comprehensive’; and,
92% ‘Agree’ or ‘Strongly Agree’ that it is ‘A useful tool’.
17. Twenty-eight comments were voiced to illuminate the raw data. Six related to whether the Guide was ‘User-friendly’: some cited the need to study the document very carefully and/or several times before it became familiar and criticised its apparent repetitive style; a better index was sought by some for rapid reference. As to being ‘Clear and unambiguous’, there were four comments indicating that it was felt to be too detailed or repetitive and used ‘vague’ terms, although the latter were not specified. To the statement, ‘Comprehensive’, there were two remarks, such as “It would help to have specific course programmes/curriculum”, indicating that the readers had misinterpreted the purpose of the document. ‘A useful tool’ brought three positive comments.
18. Thirteen general remarks included some criticism such as: “Too much like an NVQ”; “The requirements are overwhelming for a small organisation ... would need the employment of full-time trainers”; “May show areas of study and interest but has no substance”; “Far too complex and unwieldy. Produce a document written by pilots for pilots. The concept is good but please let us have something simpler to understand”; and, “It cannot be read as a ‘stand-alone document’ and must be used in conjunction with training material and syllabus to get an overall picture of the task”.
19. Finally, on a more positive note, comments were given in general support, for example: “As a CRM instructor with only 2 years’ experience, I found it very helpful and also to gain a structure in this type of training”; “An excellent resource”; and “This was a needed document to supplement or precede other sources of CRM learning publications such as government circulars or individually produced reference books, eg, The Naked Pilot”.
20. In sum, there was a sound endorsement of the practical application of the Guide in the pursuit of CRM instructor standards and competences. If it has faults, they are largely minor when compared to the benefits the product brings to practitioners.
Validity
21. As the Guide is planned to be the cornerstone of the accreditation framework, Question 11 sought to establish whether it is a ‘standard against which the competence of CRM instructors can be measured’. 38 (78%) of 49 who had seen the document were satisfied, including an academic psychologist (Table 15). One reported that more research was required before a definitive statement could be made. There were three other minor comments (given in response to Question 17) - one of which touched on the financial constraints within companies – but none negates the validity of the document.
22. The corollary to Question 11 was given in response to Question 12 (Table 16), viz, ‘I believe that CRM trainers with sufficient experience should be entitled to grandfather rights, that is, a measure of credit towards accreditation’. Not surprisingly, many who supported the Guide as an accreditation standard also sought to establish ‘grandfather rights’ for their own expertise and experience. 36 (72%) of 50 who had seen the Guide so indicated and, of all respondents, 53 (76%) of 70 supported the proposal for ‘grandfather’ rights. Comments included support for the idea that the CAA should want some evidence of prior learning/achievement and one FOI sought to require a ‘successful’ observation by a verifier. Two comments (made at Question 17) included one who did not understand the CAA’s reluctance to grant unlimited ‘grandfather’ rights to training captains as the Authority had granted such to trainers in terms of TRI ratings. He argued for companies to be permitted to monitor CRM training standards. The other respondent addressed the ‘measure of credit’, arguing that all CRM facilitators should achieve a prescribed (high) standard regardless of previous experience.
23. Among other comments made to Question 17 were those that touched on the practical guidance given in the document and the specific training required to train the trainers. Again, such respondents had not entirely grasped the purpose of the Guide but their concerns are noted and reflected further within the Recommendations.
27. Questions 15 and 16 respectively sought to identify respondents willing to contribute to the development of other guide(s) or willing to be interviewed either on the telephone or face-to-face. It was encouraging to note that 21 (54%) were willing to lend their expertise. Similarly, 62% were willing to be subject to telephone interview and 65% to meet face-to-face. Significantly, they were not always the same individuals and it was apparent that some were eager to meet for further discussion but were reluctant to have an interview on the phone.
CONCLUSIONS
28. To meet the aims of the survey, the report needed to determine: the extent of the use of the Guide; its credibility within the industry; and its validity as a tool in the accreditation of instructors of CRM training.
29. Use of the Guide is not as extensive as might be hoped given its wide distribution over a two-year period. However, the respondents who have seen and read the document are highly experienced practitioners within the industry, many in positions of authority and influence, and it is reasonable to opine that they are representative of all those who have seen and read the Guide. A significant proportion has endorsed the value of the document and criticisms are not so spectacular as to undermine its worth or credibility. On balance, the CAA can move forward with confidence to introduce the Guide as a valid keystone of the accreditation framework for instructors of CRM training.
30. Recommendations are drawn in part from the analysis of data (Appendix 2) and partly from verbatim comments, in particular those made at Questions 17 and 18. The latter sought comments concerning the direction that CRM training in general is taking in the UK, (other than validation of the Guide and CRM trainer accreditation) (see Appendix 3). Some 57% (52 respondents) provided separate comments, 75% (39) from aeroplanes and 23% (12) from helicopters. The recommendations therefore address matters beyond just the Guide. However, all remain pertinent to future CRM training in the UK and are included to inform the appropriate agencies.
31. It is recommended that the CAA notes that:
a. The results of the survey do validate the Guide as a suitable tool in the accreditation of instructors of CRM training.
b. There are genuine concerns from smaller operators, particularly with respect to the personnel resources, financial costs and potential bureaucracy required to meet the CRM accreditation proposals.
c. The CRM training available to the single-pilot community - particularly helicopter operators and including those in Public Transport - is largely based on multi-crew, fixed-wing operations. Separate CRM training material should be developed.
d. The emphasis on CRM training should be complementary to and not at the expense of technical training, especially hands-on manoeuvre training.
e. There is an apparent lack of understanding of the scope of the proposals for CRM accreditation, and some in the GA community and in the licensing regime believe they will apply equally to them. The Authority should adopt an holistic policy from ab initio training to line flying to overcome the anomalies currently inherent in MCC and CRM training. A publicity campaign – perhaps through a very widespread distribution of the forthcoming AIC – is needed to clarify precisely to whom the proposals apply. More precision is also needed with regard to ‘grandfather rights’.
f. In many parts of the industry only the introduction of Regulations will lead to the development of appropriate CRM training, including trainer training towards accreditation. Hence, the forthcoming AIC is expected to empower CRM training managers to make greater use of the Guide and to stimulate sufficient investment in the accreditation proposals.
g. There should be a pragmatic approach to integrate flight crew and cabin crew CRM training that recognises it is inappropriate to integrate all such training. Such activity should be extended to interaction with maintenance and ramp personnel.
h. There is concern that commercial video producers are marketing CRM training films without any accreditation or quality control, to which the smaller operators are particularly vulnerable.
i. To ensure the most efficient implementation of the accreditation programme, the Authority should establish and, subject to data protection/privacy laws, publish a list of the names of managers responsible for CRM training in all Public Transport AOC Holders and other training providers. It would be prudent to consider such posts as ‘named’ appointments requiring the companies to alert the CAA when post holders change. (Note: the CAA has let a contract in 2001 with the University of Aberdeen to establish a list of managers responsible for CRM training).
j. As the accreditation system is introduced, there should be data gathering of the verifier process by FOIs to validate the rigour of oversight.
32. It is recommended that the CRM Advisory Panel notes that:
a. The survey revealed that the better approach to canvass views is direct to named individuals rather than to post holders. Future mailings would therefore benefit from access to the list of CRM managers recommended at Paragraph 31i (subject to data protection/privacy laws).
b. There were many respondents who offered to assist in developing other guides - for example, for cabin crew - and who were willing to be subject to interview. The Panel should cultivate these volunteers by making contact with them in the near future.
c. The greatest weakness revealed in the Guide was that it was assessed as not user-friendly by 34% of respondents. This aspect should be addressed if the document is amended or republished and the volunteers should be approached to assist.
February 2001
Comments are welcomed to pieterhemsley@hotmail.com from whom the individual appendices listed below can be requested.
List
of Appendices:
Source Documents:
1. Guide to Performance Standards for Instructors of Crew Resource Management Training in Commercial Aviation published jointly by the RAeS, CAA and Aviation Training Association, September 1998, and printed by Riverprint, England.